Extended memory system

ABSTRACT

Described herein are technologies that are configured to assist a user in recollection information about people, places, and things. Computer-readable data is captured, and contextual data that temporally corresponds to the computer-readable data is also captured. In a database, the computer-readable data is indexed by the contextual data. Thus, when a query is received that references the contextual data, the computer-readable data is retrieved.

RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/324,838, filed on Jul. 7, 2014, and entitled “EXTENDED MEMORYSYSTEM”. The entirety of this application is incorporated herein byreference.

BACKGROUND

As computing devices have become ubiquitous, the amount of datagenerated by people has become massive, and is expected to grow nearlyexponentially. Accordingly, it is often difficult for a user to rememberwhere particular data is located (e.g., on what computing device, inwhat file, under what name), the time that the data was generated, etc.Conventionally, file systems have been used to assist users inorganizing data, and the users rely on their memory and an organizationschema (e.g., a naming convention) to locate data of interest. Thisapproach is somewhat antiquated, as many applications employed by usersgenerate data that is not well-suited for retention in manuallygenerated folders. For instance, many users employ social networkingapplications, which place data in feeds that are constantly beingupdated. Thus, a “feed” or portion thereof is typically not well-suitedfor retention in a file folder. Furthermore, the amount of data in afeed may be incredibly large, thus rendering it tedious for the user toorganize data in a manner that is different from what is supportednatively by the social networking application. Still further, users tendto employ many different types of applications, each of which retainsdata differently. Thus, it may be difficult for a user to remember whichapplication was used to generate data, what the generated datareferences, and so forth.

SUMMARY

The following is a brief summary of subject matter that is described ingreater detail herein. This summary is not intended to be limiting as tothe scope of the claims.

A computing system is disclosed herein. The computing system includes aprocessor and a memory, wherein the memory comprises an extended memorysystem that is executed by the processor. The extended memory system,when executed by the processor, is configured to receivecomputer-readable data from a first computer-executable applicationexecuting on a computing device. The extended memory system is furtherconfigured to receive contextual data that temporally corresponds to thecomputer-readable data from a second computer-executable applicationexecuting on the computing device. The extended memory system is alsoconfigured to expose the computer-readable data to a search component,the computer-readable data indexed by the contextual data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of an exemplary computing systemthat is configured to assist a user with remembering information about aperson, place, or thing.

FIG. 2 is a functional block diagram of an exemplary extended memorysystem that is configured to construct a searchable database and searchover the searchable database.

FIG. 3 is a functional block diagram of a client computing device thatis configured to assist a user with remembering information about aperson, place, or thing.

FIG. 4 is a functional block diagram of an exemplary client extendedmemory system that is configured to capture information for retention,and is further configured to facilitate retrieval of previously capturedinformation.

FIG. 5 illustrates an exemplary graphical user interface that can bepresented on a display of the client computing device that may assist auser in remembering information about a person, place, or thing.

FIG. 6 illustrates exemplary user interactions with a client computingdevice that causes the client computing device to capture informationand contextual data associated therewith.

FIG. 7 illustrates an exemplary graphical user interface thatfacilitates inviting an extended memory system as a contact in a socialnetworking application, wherein the extended memory system is configuredto capture content generated by way of the social networking applicationas well as context corresponding to such content.

FIG. 8 illustrates an exemplary graphical user interface of a unifiedcommunications (UC) application, where an extended memory system hasbeen invited to a meeting conducted by way of the UC application.

FIG. 9 illustrates an exemplary graphical user interface of an e-mailapplication, wherein the e-mail is transmitted to an extended memorysystem for retention and indexing.

FIG. 10 illustrates an exemplary graphical user interface of a contentpage, where content displayed on the content page is selected and causedto be stored and indexed in a database.

FIG. 11 illustrates an exemplary graphical user interface thatfacilitates disambiguating an entity during content creation.

FIG. 12 is a flow diagram that illustrates an exemplary methodology forindexing computer-readable data in a database by contextual datacorresponding to the computer-readable data.

FIG. 13 is a flow diagram that illustrates an exemplary methodology forindexing computer-readable data by contextual data correspondingthereto.

FIG. 14 is a flow diagram that illustrates an exemplary methodology fordisambiguating an object or entity referenced in computer-readable dataand indexing the computer-readable data by the disambiguated object orentity.

FIG. 15 is a flow diagram that illustrates an exemplary methodology forperforming a search over a database, wherein a query employed to performthe search comprises a reference to contextual data.

FIG. 16 is a flow diagram that illustrates an exemplary methodology forautomatically constructing a query to execute over a database based uponan identified context of a user.

FIG. 17 is a flow diagram that illustrates an exemplary methodology forgenerating and transmitting contextual data to be used to index capturedcomputer-readable data.

FIG. 18 is an exemplary computing system.

DETAILED DESCRIPTION

Various technologies pertaining to assisting users recollect informationabout people, places, and/or things are now described with reference tothe drawings, wherein like reference numerals are used to refer to likeelements throughout. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of one or more aspects. It may be evident,however, that such aspect(s) may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to facilitate describing one or moreaspects. Further, it is to be understood that functionality that isdescribed as being carried out by certain system components may beperformed by multiple components. Similarly, for instance, a componentmay be configured to perform functionality that is described as beingcarried out by multiple components.

Moreover, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B. In addition, the articles “a” and“an” as used in this application and the appended claims shouldgenerally be construed to mean “one or more” unless specified otherwiseor clear from the context to be directed to a singular form.

Further, as used herein, the terms “component” and “system” are intendedto encompass computer-readable data storage that is configured withcomputer-executable instructions that cause certain functionality to beperformed when executed by a processor. The computer-executableinstructions may include a routine, a function, or the like. It is alsoto be understood that a component or system may be localized on a singledevice or distributed across several devices. Further, as used herein,the term “exemplary” is intended to mean serving as an illustration orexample of something, and is not intended to indicate a preference.

With reference now to FIG. 1, air exemplary computing system 100 that isconfigured to assist users in recollecting information about people,places, and/or things is illustrated. The computing system 100 may be asingle computing device or a distributed computing system. For example,the computing system 100 may be a rack-mounted server in a data center,several rack-mounted servers in the data center, a portion of anenterprise computing system, an entirety of the data center, or thelike. The computing system 100 includes a processor 102 and a memory104, wherein contents of the memory 104 are accessible to the processor102. With more particularity, the memory 104 includes an extended memorysystem 106 that is executable by the processor 102. The computing system100 additionally includes a data store 108 that includes a searchabledatabase 110. As will be described in greater detail herein, thedatabase 110 includes computer-readable data generated by applicationsemployed by users, and further includes contextual data corresponding tosuch computer-readable data. For instance, the computer-readable datamay be indexed in the database 110 by the contextual data.

Operation of the computing system 100 will now be generally described.As indicated previously, the computing system 100 is a client computingdevice employed by a user or is in communication with the clientcomputing device. The client computing device can have applications thatare executable thereon, wherein the applications are configured togenerate computer-readable data based upon input from the user, wherethis input can be provided explicitly by the user or implicitly. Forexample, an application can monitor location of a user and logactivities of the use—thus, the user need not explicitly provide suchinformation. In another example, an operating system or application cantrack which applications are being used by the user and the associatedcontexts. Collection of such data, for instance, can be preconfigured.The applications may also be configured to generate computer-readabledata for presentment to the user. At least some of the computer-readabledata generated by the applications may desirably be obtained forsubsequent user review, for issuing a reminder, for generating arecommendation, etc. Accordingly, computer-readable data 112 generatedby an application on the client computing device can be received at thecomputing system 100 for storage and indexing.

In connection with receiving the computer-readable data 112, the user ofthe client computing device may optionally set forth an indication thatthe computer-readable data 112 is to be retained, thereby allowing forsubsequent retrieval and/or analysis of the computer-readable data 112.Exemplary mechanisms for indicating that computer-readable data is to beretained are set forth below. Additionally, contextual data 114 thattemporally (and optionally spatially) corresponds to thecomputer-readable data 112 can be received from the client computingdevice or from some other device (e.g., a device used to support a webservice). For instance, current weather conditions can be received froma computing device that supports a web service that is configured toprovide updated weather information. The contextual data 114 isdescriptive of a context corresponding temporally to thecomputer-readable data 112. In an example, the computer-readable data112 can be generated by a first computer-executable application and thecontextual data 114 can be generated by a second computer-executableapplication.

In a non-limiting example, a user may be viewing a web page and may reada story that the user finds interesting; thus the user may wish toremember at least a portion of the story. The user can set forth anindication that the story on the web page is desirably remembered by theuser (e.g., by tapping on a touch-sensitive display, through selectionof a menu option that is presented upon right-clicking, etc.), which cancause the computer-readable data 112 to be provided to the computingsystem 100. In this example, the computer-readable data 112 can includethe story itself, a link to the web page, a portion of the story, or thelike. Additionally, at least one other application executing on theclient computing device can capture contextual data that temporallycorresponds to the user viewing the web page. For example, thecontextual data may be identities of other applications that are beingexecuted when the user is viewing the web page, an identity of a personwith whom the user is meeting when the user is viewing the web page, anevent occurring in the life of the user when the user is viewing the webpage (e.g., the user is viewing the web page on her birthday), a weatherevent that is occurring when the user is viewing the web page (e.g., athunderstorm, a blizzard, a temperature outside), a news event thatoccurs on the day that the user is viewing the web page, etc.

The extended memory system 106 receives the computer-readable data 112and the contextual data 114 and indexes the computer-readable data 112in the database 110 by the contextual data 114. In another example, thecomputer-readable data 112 can be indexed by the content of thecomputer-readable data. The extended memory system 106 can receivemultiple instances of computer-readable data and respectivecorresponding contextual data over time, and can populate the database110 with such computer-readable data and corresponding contextual data,where the computer-readable data is indexed by the contextual data.

When formatted in this manner, the database 110 supports retrievalof: 1) computer-readable data based upon contextual data correspondingthereto; 2) contextual data based upon computer-readable datacorresponding thereto; and 3) retrieval of computer-readable data and/orcontextual data based upon a combination of computer-readable data andcontextual data. For example, the extended memory system 106 canretrieve appropriate computer-readable data in response to queries suchas “what web page was I looking at during my meeting with John Doe”,“who came to my birthday party last year”, “what was I doing when thepresident was inaugurated”, “what story was I reading when the song Xwas playing on my device”, “find the page I was reading last week aboutCompany A”, amongst others.

As will be described in greater detail herein, a query to be executedover the database 110 can be received from a client computing deviceemployed by a user, can be automatically constructed based uponcontextual data (e.g., to generate recommendations or reminders), or canbe a disambiguated query, wherein the user sets forth a query and theextended memory system 106 disambiguates the query. This may occur whena query references an ambiguous entity, and the extended memory system106 can add metadata to the query or reformulate the query such that theentity is disambiguated. Due to the volume of contextual data that isretained, searches for computer-readable data can be completed morequickly when compared to conventional mechanisms for locatingcomputer-readable data, which often include multiple searches,navigation through file folders, etc. For example, utilizing theextended memory system 106, a user can set forth the query “whatdocument was 1 editing when meeting with Jane?”, and the extended memorysystem 106 can execute a search using the contextual information aboutJane, causing the document being searched for to be retrieved relativelyquickly. In contrast, without the extended memory system 106, the userwould be forced to attempt to recollect a word or phrase in thedocument, and search over documents contents (particularly if there area large number of documents) may take a significant amount of time.

Now referring to FIG. 2, a functional block diagram of the extendedmemory system 106 is illustrated. The extended memory system 106comprises a database constructor component 202 that is configured tobuild and populate the database 110. The extended memory system 106 alsoincludes a search component 204 that is configured to execute searchesover the database 110. The database constructor component 202 and thesearch component 204 will be discussed in turn.

The database constructor component 202 comprises an entity disambiguatorcomponent 206 that is configured to disambiguate entities referenced inthe computer-readable data 112 that is to be indexed by the databaseconstructor component 202. For example, computer-readable data 112 mayinclude reference to an entity or entities that may be non-uniform orsomewhat ambiguous. For instance, the computer-readable data 112 mayinclude the acronym “US”, which may refer to the United States,ultrasound, under study, user segment, amongst others. In anotherexample, computer-readable data 112 to be indexed by the databaseconstructor component 202 may include the word “bears” where, dependingupon context, the term “bears” may refer to a professional United Statesfootball team, a collegiate mascot, stocks, animals, etc.

The entity disambiguator component 206 may utilize a variety ofapproaches to disambiguate entities referenced in receivedcomputer-readable data. In an example, the entity disambiguatorcomponent 206 can review historic observations about actions of the userto identify topics of interest to the user, and can disambiguateentities based upon identified topics of interest. For instance, if theuser is known to be an avid sports fan (e.g., based upon previouscomputer-readable data indexed by the database constructor component202), the entity disambiguator component 206 can infer that the term“bears” in computer-readable data likely refers to a professionalfootball team. In another example, the entity disambiguator component206 may include or have access to a dictionary (for the user) that mapswords or word sequences to a disambiguated entity. Thus, the term“bears” may be included in the dictionary and mapped to “Chicago Bears”.

In yet another example, the entity disambiguator component 206 canutilize natural language processing (NLP) techniques to identify acontext in which an ambiguous term appears, and can disambiguate theterm based upon such context. Therefore, for instance, the entitydisambiguator component can disambiguate the term “bears” in thecomputer-readable data as referring to an animal when other portions ofthe computer-readable data refer to “carnivorous mammals”. In situationswhere the computer-readable data represents a conversation, the entitydisambiguator component 206 can disambiguate pronouns in theconversation. For instance, the computer-readable data may include thephrase “I like the shirt that you are wearing”. In such phrase, whenviewed in isolation, the pronoun “you” is ambiguous. The entitydisambiguator component 206, however, may disambiguate such pronoun byreceiving and analyzing other contextual data. For instance, the entitydisambiguator component 206 can scrape a calendar of the user, anddetermine that the user is scheduled to meet with John Doe when thephrase was uttered. Thus, the entity disambiguator component 206 maydisambiguate the pronoun “you” as referring to John Doe.

The entity disambiguator component 206 may further utilize voicerecognition technology to determine identity of a speaker, which in turncan be used to disambiguate a pronoun in a conversation. In still yetanother example, the entity disambiguator component 206 can utilizelocation to disambiguate an entity. For instance, a user may capture animage of a football stadium using a mobile telephone, and may tag thephoto with the metadata “great day to watch a game”. Here, the term“game” is ambiguous. The entity disambiguator component 206 candisambiguate the term “game” based upon time that the photo was capturedand location where the photo was captured. For instance, using thelocation and time, the entity disambiguator component 206 can identify agame that is scheduled to occur at the location and time. The entitydisambiguator component 206 may then assign metadata to the image thatdisambiguates the term “game”.

Still further, the entity disambiguator component 206 may employvision-based technologies to disambiguate an entity and/or object incomputer-readable data that is to be indexed by the database constructorcomponent 202. For example, the user may be participating in a videoconference with a particular person, where the name of the person isnever spoken in the conversation. The user may wish to recollect atleast a portion of the conversation, such that the portion of theconversation is to be indexed in the database 110 by the databaseconstructor component 202. The entity disambiguator component 206 canreceive an image of a face of the person, and can use face recognitiontechnologies to identify the person. The entity disambiguator component206 can apply the identity of the person as metadata to thecomputer-readable data. Thus, if the user subsequently queries thedatabase 110 using the identity of the person, the appropriate videoconference can be identified and returned.

In summary, then, the entity disambiguator component 206 can receivecomputer-readable data that is to be included in the database 110. Theentity disambiguator component 206 can disambiguate objects and/orentities referenced in the computer-readable data and can generatemetadata for the computer-readable data based upon disambiguation of theobjects and/or entities. The computer-readable data may then be indexedby the database constructor component 202 by the metadata. It can beascertained that the entity disambiguator component 206 can beconfigured to generate this metadata in a uniform manner, such that, forinstance, US, USA, US of A, and United States may all map to thedisambiguated entity the United States of America.

The database constructor component 202 can also include a contextacquirer component 208 that is configured to acquire contextual dataabout received computer-readable data. For example, responsive to thedatabase constructor component 202 receiving computer-readable data, thecontext acquirer component 208 can ascertain contextual datacorresponding thereto. If there is insufficient contextual data or ifmore contextual data is desired, the context acquirer component 208 canacquire contextual data about the received computer-readable data. Forinstance, the context acquirer component 208 can cause a signal to betransmitted to a client computing device that requests contextcorresponding to the client computing device. This context may be thelocation of the client computing device, identities of applicationsbeing executed on the client computing device, people in proximity tothe client computing device, contextual data logged over time (e.g.,over a specified time window), or the like. The database constructorcomponent 202 can then index the computer-readable data by thecontextual data. In the example where context is logged over time, aquery over the database 110 using context results in filters beingapplied by time.

As indicated previously, the extended memory system 106 is configured tobetter lives of people by assisting them with recollecting informationabout people, places, things, or the like. That is, the extended memorysystem 106 is configured to improve lives of people through use oftechnology. To that end, the search component 204 is configured toexpose the computer-readable data to search, wherein the database 110can be searched based upon contextual data corresponding tocomputer-readable that may be remembered by the user, rather thancontent of the computer-readable data itself. Again, as noted above, thesearch component 204 can support searching over the database 110 basedupon a query such as “what documents were discussed during a meeting Ihad with John Doe”.

To facilitate searching over the database 110, the search component 204can include a query disambiguator component 210 that can disambiguateentities and/or objects in a query received from a client computingdevice employed by the user. With more particularity, similarly to thecomputer-readable data described above, a query may include ambiguouskeywords. The query disambiguator component 210 may utilize a variety oftechnologies to disambiguate entities and/or objects referenced in thequery. For instance, the query disambiguator component 210 can receiveprevious queries issued by the user and disambiguate the received querybased upon the previous queries (e.g., queries related to a particulartopical interest in the past indicate that the user in will continue tobe interested in the topic). In another example, the query disambiguatorcomponent 210 can receive identities of search results selected by theuser, wherein the search results have categorical (contextual) dataassigned thereto, for instance, by way of the Open Directory Project(ODP). This allows for the query disambiguator component 210 todetermine categories or topics that may be of interest to the user,which can be used to disambiguate the query and/or terms therein. Inanother example, the query disambiguator component 210 can access thedatabase 110 and identify categories assigned to entries in the database110. In a non-limiting example, the query disambiguator component 210can search the database 110 based upon an ambiguous object and/or entityin the query, and can identify at least one disambiguated version of theobject and/or entity in the database 110. The query disambiguatorcomponent 210 can disambiguate the keyword or phrase in the query basedupon frequently occurring disambiguations of the keyword or phrase.

In a non-limiting example, the query disambiguator component 210 canreceive the query “who was I with when watching the bears”. Based uponprevious interests exhibited by the user (e.g., determined based uponexplicit user input or determined through inference based upon observedprevious actions of the user), the query disambiguator component 210 candetermine that the user has an interest in nature, and thus, the term“bears” most likely refers to an animal, rather than a collegiate team.The query disambiguator component 210 may then add metadata to the querythat indicates that the term “bears” refers to an animal, and the searchcomponent 204 can search the database 110 based upon the metadata. Inanother example, the search component 204 can perform a search thatcovers numerous (e.g., all) interpretations of an ambiguous query, andperform filtering of results on the back-end based upon explicitlyprovided or learned preferences. Referring to the example above, evenwhen the user is interested in nature, but was not with anyone whenwatching bears (but did attend a collegiate sporting event with “Bears”as a mascot), a person with whom she attended the sporting even can bereturned as a potential answer.

The search component 204 can also include a recommender component 212that is configured to search the database 110 and output recommendedcomputer-readable data to a user. In an example, the recommendercomponent 212 can receive a context related to the user and canconstruct a query based upon the context. For instance, the recommendercomponent 212 can receive an indication that the user has a meetingscheduled for a certain time, where the term “bears” is included in thesubject line of the meeting. The recommender component 212, based uponthis context, can construct a query that includes the keyword “bears”,and the search component 204 can search the database 110 forcomputer-readable data pertaining to “bears”. The recommender component212 can utilize temporal information (context) to determine when tooutput a recommendation to the user (e.g., when to providecomputer-readable data as a recommendation). For instance, if themeeting is at a particular time, the recommender component 212 canconstruct the aforementioned query and cause the query to be executed 15minutes prior to the meeting. The recommender component 212 can thenoutput computer-readable data (or links to the computer-readable data)located based upon the query as a recommendation. In this example, theuser can enter the meeting fully prepared with computer-readable datathat is pertinent to the meeting. Thus, the recommender component 212can receive contextual data about the user (e.g., identities ofapplications the user is currently employing, location of the user, whothe user is with, a schedule of the user, etc.), and can constructqueries based upon the contextual data. The search component 204 canexecute a search based upon the constructed queries over the database110, and can output computer-readable data as recommendations to theuser.

The search component 204 also includes a reminder component 214 that canexecute searches over the database 110, wherein the queries aretriggered based upon contextual data. The contextual data may betemporal, location-based, event-based, or the like. For instance, theuser may wish to be reminded to stop at the grocery store after workwhen traveling close to a grocery store, and may register a reminderwith the extended memory system 106. The reminder can be monitored bythe reminder component 214, such that when the context set forth in thereminder is triggered, the reminder component 214 can execute a queryover the database 110 and output corresponding computer-readable data(or links thereto) as a reminder to the user. In another example, theuser may be provided with a computer-implemented advertisement thatindicates that a retail store is having a sale on a product of interestto the user. The user may set forth an indication that is desirable thatthe sale be retained in the database 110 as something that the userwishes to remember. The user can add metadata to the advertisement, suchas location of the retail store (or such location can be automaticallydetermined by the context acquirer component 208). Additional contextualinformation may include when the sale is active, and a proximity to thestore of the user. The reminder component 214 can monitor context of theuser over time and can generate queries based upon context of the user.When context of the user matches the reminder registered with theextended memory system 106, the reminder component 214 can cause thesearch component 204 to execute a search over the database 110, toretrieve the computer-readable data, and present the computer-readabledata to the user as a reminder. Again, the extended memory system 106assists the user in recollecting something that the user was interestedin recollecting.

In another example, the reminder component 214 can be used to help theuser remind someone else of a task or event. In such a scenario, theuser can identify a person or persons who are to be reminded e.g.,“remind John to pick up milk if he passes the grocery store.” The usercan further specify how John is to be reminded. For example, the usercan select a particular application that John is known to use, whereinthe extended memory system 106 can retain account information for John,including an identity of the application and a username for John. Thisreminder can be retained in the database 110, and the reminder component214 can query the database using contextual information about John”. Forexample, John can consent to allow the extended memory system 106 toacquire certain contextual information about John, including hislocation over time. The reminder component 214 can query the database110 based upon contextual information about John, and can provide thereminder to John when the reminder is triggered.

In still yet another example, continuing reference to the exemplaryscenario set forth above, when the user identifies John as the person tobe reminded (e.g., by way of the extended memory system 106), John maybe provided with a reminder request from John's extended memory system,which originated from the extended memory system 106 of the user. Thus,the extended memory system 106 of the user may communicate with otherextended memory systems offered by the same service. Alternatively, Johnmay be provided with an email, a text message, or other electronicmessage, which includes the reminder request. When John accepts thereminder request, an extended memory system may be set up for John (ifnot already existent). The reminder request can indicate that the userwishes to remind John about something (e.g., to pick up milk). When Johnaccepts the request, the extended memory system for John can monitorcontext about John, and can trigger the reminder when appropriate. Uponcompletion of the task referenced above (John has picked up milk at thegrocery store), the extended memory system for John can optionallyinform the extended memory system 106 for the user that the task hasbeen completed.

Further, the user can specify more granular information about wheninformation is to be shared by way of the reminder component 214. Forinstance, referring to the example above, the user can indicate thatJohn is to be reminded only if John is near the grocery store within atime window or time windows. For example, the user can specify that Johnis only to be reminded if passing the grocery store sometime within thenext three days, between the hours of 5:00 pm and 6:00 pm, and only iftraffic is not congested.

Still further, the reminder component 214 can help a user with sharinginformation with a certain person or persons. This sharing ofinformation can be considered a reminder regarding who to shareinformation with, when to share the information, and how to share theinformation. For instance, the user can capture an image using theirmobile telephone, and can indicate that the image is to be shared withfamily members if the family members happen to travel to a locationproximate to the location where the image was captured. Further, theuser can indicate how this information is to be shared—e.g., by way of asocial networking application, an SMS message, an email message, or thelike. The family members of the user can optionally register with theextended memory system 106, such that contextual information about thefamily members can be tracked over time, and the reminder component 214can use such contextual information to query the database 110. Inanother example, the user can identify account information about anotheruser; e.g., a social networking account, an email account, a telephonenumber, or the like. The user may wish to share computer-readable datawith the another user (e.g., an image, a blog post, etc.), and caninstruct reminder component 214 to provide the information to theanother user when contextual data specified by the user triggers sharingof the computer-readable data. For example, the user may wish to providea coupon for suntan lotion to the another user, wherein such coupon isdesirably surfaced to the another user (e.g., by way of an applicationspecified by the user) when the temperature is above 70 degrees(Fahrenheit) at a certain location. The user may further specify anexpiration date for the coupon. The reminder component 214 can cause thecoupon to be provided to another user when this contextual informationtriggers the reminder (e.g., to share the coupon).

The search component 204 can also optionally include a dialog engine 216that is configured to enable a continuous dialogue to be undertakenbetween the user and the extended memory system 106. For example, theuser may initially set forth a query, and the search component 204 canreceive the query and identify a plurality of results from the database110 based upon the query. The dialog engine 216 can be configured tooutput at least a subset of the search results, as well as set forthsuggestions to the user as to other items that may be of interest to theuser. The user may then “search by browsing”—for instance, the user canselect a suggestion, which can act as a filter over the retrieved searchresults or act as a new query.

For example, the user may set forth the query, “who was the opening bandat the concert I went to with John Doe?” The search component 204 canexecute a search over the database 110 based upon the query and canoutput computer-readable data based upon the query (e.g., forpresentment to the user). The dialog engine 216 may also outputsuggestions to help the user browse through what may be of interest tothe user. For example, the dialog engine 216 can allow the user to queryby “John Doe” only, such that additional computer-readable data indexedby “John Doe” in the database 110 can be retrieved. In another example,the computer-readable data can indicate that the band “band” was theopening act at the concert that the user went to with John Doe. Thedialog engine 216 can highlight “band” in the search results, and canalso highlight other people who were at the concert with the user. Theuser may then select these other people, which can cause queries to beconstructed based upon the people selected by the user. Thus, the dialogengine 216 facilitates exploration of content of the database 110through browsing.

Now referring to FIG. 3, an exemplary client computing device 300 thatis configured to capture: 1) computer-readable data; and 2) temporallycorresponding contextual data is illustrated, wherein such data iscaptured for inclusion in the database 110. The computing device 300 maybe a mobile computing device (such as a mobile telephone, a tabletcomputing device, a wearable computing device, etc.), a laptop computingdevice, a desktop computing device, etc. The client computing device 300includes a processor 302 and a memory 304 that is accessible by theprocessor. The memory 304 includes a client extended memory system 306that is configured to capture computer-readable data and correspondingcontextual data for inclusion the database 110. The memory 304 alsoincludes a plurality of computer-executable applications 308-310. Theapplications 308-310 can include word processing applications, webbrowsing applications, gaming applications, or any other suitableapplication that can be executed by a client computing device 300. Theclient computing device 300 also includes a plurality of sensors312-314. The sensors 312-314 can include a microphone, a gyroscope, aspeedometer, an accelerometer, a positioning sensor (e.g., a GPSsensor), a camera, a thermometer, a barometer, or the like.

As indicated previously, the client extended memory system 306 isconfigured to capture computer-readable data (e.g., generated by firstapplication 308), is further configured to capture context temporallycorresponding to the computer-readable data (e.g., where the context canbe generated by the nth application 310). The client extended memorysystem 306 can capture data responsive to receipt of various usercommands. For example, a user can set forth a voice command that causesthe client extended memory system 306 to capture computer-readable dataand corresponding contextual data. In another example, the clientextended memory system 306 can capture computer-readable data andcorresponding contextual data responsive to receipt of a gestureperformed on a touch sensitive display of the client computing device300. In yet another example, the client extended memory system 306 canbe configured to capture computer-readable data and correspondingcontextual data based upon a signal output by one or more the sensors312-314. For instance, when a positional sensor indicates that the userhas remained stationary for a relatively long period of time, and avoice sensor indicates that the user is having a conversation withanother person, the client extended memory system 306 can inter that theuser may wish to remember at least portions of the conversation and cancause computer-readable data to be captured (e.g., can cause atranscription of the conversation to be generated) and can further causecorresponding contextual data to be captured (e.g., time of theconversation, location where the conversation occurred, person with whomthe user was having the conversation, ambient temperature when theconversation was occurring, world news events happening when theconversation was occurring, etc.).

With reference now to FIG. 4, a functional block diagram of the clientextended memory system 306 is illustrated. The client extended memorysystem 306 includes a setup component 402 that is configured tofacilitate setting up an account with the extended memory system 106 forthe user of the client computing device 300. The setup component 402 canreceive a user identifier, as well as information as to what type ofinformation can be collected when the user employs the client computingdevice 300 or other computing device that is to be associated with theaccount (e.g., a device upon which the user can provide logincredentials). For example, the user can identify applications from whichcomputer-readable data can be collected, and can identify applicationsfrom which computer-readable data is not to be collected. Furthermore,by way of the setup component 402, the user can set forth restrictionsas to times that computer-readable data can be collected, locations werecomputer-readable data can and cannot be collected, computing devicesfrom which data can and cannot be collected, etc.

The client extended memory system 306 can also include a client capturecomponent 404 that is configured to capture computer-readable dataresponsive to the user of the client computing device 300 setting fortha command and/or responsive to a particular context being observed.Exemplary commands that can be recognized by the content capturecomponent 404 that cause the content capture component 404 to capturecomputer-readable data include a gesture on a touch-sensitive display(e.g., a tap on a touch sensitive display), a voice command, a gesturein space that can be recognize by analyzing video of the user, selectionof a menu item, selection of a graphical button, etc. In anotherexample, the content capture component 404 can be configured to capturecomputer-readable data responsive to a predefined context beingrecognized. For instance, in an example, the content capture component404 can be configured to capture particular computer-readable data eachtime the user is at a particular location. As described previously,computer-readable data captured by the content capture component 404 canbe transmitted by the client extended memory system 306 to the computingsystem 100, where the extended memory system 106 can store or referencethe computer-readable data in the database 110.

The client extended memory system 306 also includes a context capturecomponent 406 that is configured to capture contextual data thattemporally corresponds to the computer-readable data. Exemplarycontextual data that can be captured by the context capture component406 may include, but is not limited to, identity of an application fromwhich the computer-readable data is captured, other applicationsexecuting at the time that the computer-readable data was captured, anidentity of a person with whom the user of the client computing device300 is known to be with when the computer-readable data is captured,time of day that the computer-readable data is captured, weatherconditions when the computer-readable data was captured, season (e.g.,spring, summer, fall, or winter) when the computer-readable data wascaptured, prominent news events when the computer-readable data wascaptured, events in life specific to the user when the computer-readabledata was captured (e.g. birthday of the user, family members of theuser, friends of the user, an anniversary, etc.), and so forth.

With respect to identifying a person or people in proximity to the userwhen the computer-readable data is captured by the content capturecomponent 404, the context capture component 406 can identify the personor people in a variety of manners. In a first example, the contextcapture component 406 can scrape a calendar application of the user ofthe client computing device 300 and, for a time corresponding to whenthe computer-readable data is captured, identify people referenced inthe calendar application. In another example, hardware of the clientcomputing device 300 can be leveraged to identify people with whom theuser of the client computing device 300 is interacting when thecomputer-readable data is captured. For instance, the sensors 312-314 ofthe client computing device 300 may include a camera that capturesimages of the surroundings of the client computing device 300. Imageanalysis software can be configured to search the image for faces, andfacial recognition technologies can be used to identify a personcaptured in an image. In another example, the sensors 312-314 caninclude a microphone, and the client computing device 300 may leveragevoice recognition technologies that can identify a person based upon hervoice. In still yet another example, the sensors 312-314 may include aproximity sensor that can detect when the client computing device 300 isin proximity to another client computing device of another user (e.g.,through a handshake protocol). In such an example, the client computingdevices can transmit data therebetween that is indicative of identitiesof users of the devices. Moreover, the user of the client computingdevice 300 may explicitly identify the person by way of a voice command,by way of entry of an identity of the person through a keyboard, etc.Responsive to the content capture component 404 capturing thecomputer-readable data and the context capture component 406 capturingcontextual data about the computer-readable data, the extended memorysystem 306 can be configured to transmit such data to the computingsystem 100.

The client extended memory system 306 further includes a contentretrieval component 408 that is configured to facilitate retrieval ofdata from the database 110 of the computing system 100. The contentretrieval component 408 can include an input receiver component 410 thatreceives input relevant to data in the database 110. In an example, theinput receiver component 410 can receive a query explicitly set forth bythe user, wherein the query, for example, may include a reference to thecontextual data that is used to index the computer-readable data in thedatabase 110. The extended memory system 106 can retrieve relevantcomputer-readable data based upon the query, wherein conventionallyqueries that reference contextual data are not readily supported. Thequery received by the input receiver component 410 may be a voice queryset forth by a user, a query set forth by way of a keyboard, or othersimilar mechanism for inputting a query. In another example, the inputreceiver component 410 can receive contextual data output by one or moreof the applications 308-310 installed on the client computing device 300or from one or more of the sensors 312-314 on the client computingdevice 300. The input receiver component 410, responsive to receipt ofsuch contextual data, can construct a query based upon the contextualdata and transmit the query to the extended memory system 106 of thecomputing system 100.

The content retrieval component 408 also includes an output component412 that receives data identified by the extended memory system 106 asbeing relevant to the query, where the query was transmitted to theextended memory system 106 by the input receiver component 410. Forexample, the output component 412 can output search results to a displayof the client computing device 300, where the search results includecomputer-readable data in the database 110, identifiers forcomputer-readable data in the database 110, links to computer-readabledata in the database 110, etc. The output component 412 may alsogenerate audio output pertaining to search results. For example, whenthe search result includes a computer-readable data that represents acaptured conversation, the output component 412 can replay the relevantportion of the conversation. Likewise, the output component 412 canpresent a video, an image, or the like.

In an example, the client extended memory system 306 can be included asa portion of the operating system for the client computing device 300.That is, the client extended memory system 306 can be built into theoperating system of the client computing device 300 and, therefore, canhave access to output of the applications 308-310 installed on theclient computing device 300 and/or the sensors 312-314 on the clientcomputing device 300. In another example, the client extended memorysystem 306 may be a standalone application installed on the clientcomputing device 300, and one or more of the application 308-310 can beconfigured to transmit data to such application. Furthermore, while theclient extended memory system 306 and the extended memory system 106 hadbeen referenced as being included in separate computing systems, it isto be understood that the client extended memory system 306 and theextended memory system 106 may reside in a same computing system. Forexample, the client computing device 300 may be configured with theextended memory system 106. Moreover, the client extended memory system306 may be configured with functionality described as being performed bythe extended memory system 106. For instance, the client extended memorysystem 306 may be configured with disambiguation functionality, such asthat described above with respect to the entity disambiguator component206 and the query disambiguator component 210. In yet another example,the client extended memory system 306 can be included as a portion of anetwork-accessible service, such as one that acts to forward messages toan agent email address, invite an agent to a chat, etc.

Now referring to FIG. 5, an exemplary graphical user interface 500 thatfacilitates capturing computer-readable data for transmittal to theextended memory system 106 is illustrated. Thus, the graphical userinterface 500 can be presented on a display of the client computingdevice 300. The graphical user interface 500 includes a content portion502 that depicts content. The content may be text, images, a video, etc.The graphical user interface 500 also includes a field 504 that includesseveral selectable buttons 506-512. In an example, the field 504 can beinvoked responsive to the user setting forth a command on the clientcomputing device 300. In a non-limiting example, the field 504 can beinvoked by way of a gesture set forth by the user of the clientcomputing device 300, such as a swipe from the bottom of a displayupwards, a swipe from a side of the display inwards, etc. In anotherexample, the field 504 can be invoked responsive to receipt of a voicecommand from the user of the client computing device 300. At least oneof the buttons 506-512 in the field 504, when selected, can cause atleast a portion of the content shown in the content field 502 to becaptured, and can further cause contextual data corresponding thereto tobe captured (as described above). The content and correspondingcontextual data can then be transmitted to the computing system 100,where the extended memory system 106 may then index thecomputer-readable data in the content field 502 in the database 110 bythe contextual data.

Now referring to FIG. 6, an exemplary mobile computing device 600 isillustrated. The mobile computing device 600 may be a mobile telephone,a tablet computing device, a wearable computing device (such as awatch), or the like. The mobile computing device 600 includes a touchsensitive display 602. The touch sensitive display 602 may presentcontent thereon. A user 604 may perform a gesture relative to the touchsensitive display 602 to cause the content shown thereon to be capturedand transmitted to the extended memory system 106 together withcorresponding contextual data. In an example, the user 604 may “tap” thetouch sensitive display 602 with a member 606, such as a finger, astylus, or the like. The content capture component 404 of the clientextended memory system 306 in the mobile computing device 600 can detectthe “tap” and can cause content shown on the touch sensitive display 602to be captured and transmitted to the computing system 100.Additionally, the client computing device 600, responsive to the contentcapture component 404 detecting the “tap”, can cause contextual data tobe captured and transmitted to the computing system 100. As indicatedpreviously, in addition to or alternatively to capturing content andcontextual data responsive to user interaction with respect to the touchsensitive display 602, the mobile computing device 600 can be configuredto capture content shown on the touch sensitive display 602, as well ascorresponding contextual data responsive to identifying some otherinput, such as a spoken utterance set forth by the user 604, a gesturemade by the user 604 (e.g., a wink or series of winks), etc.

Now referring to FIG. 7, an exemplary graphical user interface 700 of asocial networking application is shown. The graphical user interface 700includes a feed that can be presented to the user, where the feedincludes first content 702, second content 704, and third content 706.The content 702-706 may be posts made by contacts of the user, postsmade by the user, posts made by someone who the user is following, andso forth. The user may wish that the extended memory system 106 beconfigured to obtain computer-readable data in the social networkingapplication, and index the computer-readable data by contextual datacorresponding thereto. Accordingly, for instance, the graphical userinterface 700 may include a button 708 that can be selected to invite acontact to be a “friend” on the social networking application. In theexample shown in FIG. 7, the user may choose to invite a representationof the extended memory system 106 (shown in window 710). Accordingly,the user has authorized the extended memory system 106 to monitorinformation presented in the social networking application to which theuser has access. The user may set forth more granular information as towhat is to be captured by the extended memory system 106, such asidentities of contacts of the user whom the extended memory system 106is to follow (e.g., such that posts generated by the identified contactscan be captured and indexed), types of content the user wishes for theextended memory system 106 to capture (e.g., images and videos, but nottext), and so forth.

Now referring to FIG. 8, another exemplary graphical user interface 800that can be presented on a display of the client computing device 300 isillustrated. The graphical user interface 800 may be for an instantmessaging application, a unified communications (UC) application, avideoconferencing application, etc. In the exemplary graphical userinterface 800, a content field 802 may include content presented in ameeting by one or more participants of the meeting. A plurality ofgraphical objects 804-810 respectively represent participants in themeeting. The graphical user interface 800 may further include a textentry field 812, wherein the user can set forth textual information tomeeting participants by way of the text entry field 812. In thisexample, the user may request assistance in remembering contents of themeeting. Accordingly, the user can invite a representation of theextended memory system, as shown by the graphical object 810. Thus,other participants will be notified that the meeting is being monitored,and content set forth during the meeting is being provided to theextended memory system 106. Audio of the meeting can be captured by theclient extended memory system 306. In this example, the client extendedmemory system 306 can be included in a data center that hosts theapplication; since the audio is shared with the agent that executes inthe data center, the client computing device need not have to record themeeting locally. The client extended memory system 306 can provide theaudio to the extended memory system 106, may generate a transcription ofthe audio and provide the transcription to the extended memory system106, can capture content in the content field 802 and transmit suchcontent to the extended memory system 106, etc., can capture identitiesof the meeting participants and transmit such identities to the extendedmemory system 106, etc. Further, the client extended memory system 306can capture contextual data about the meeting and transmit suchcontextual data to the extended memory system 106, which can indexportions of the meeting by the contextual data.

With reference now to FIG. 9, an exemplary graphical user interface 900of an e-mail application is illustrated. The graphical user interface900 includes a content field 902, where the user can review content orenter a message that is to be transmitted by way of an e-mailapplication. The graphical user interface 900 also includes a “To” field904, where the user can enter e-mail addresses of intended recipients ofthe e-mail. In the example shown in FIG. 9, the user wishes to send thee-mail to the e-mail address “e-mail@e-mail.com.” The user can alsoindicate that she wishes for the extended memory system 106 to indexcontent of the e-mail by entering an e-mail address for the extendedmemory system 106 in the “To” field. For example, the e-mail address“memory@memory.com” may be an e-mail address assigned to the extendedmemory system 106, such that the extended memory system 106 knows thatthe user wishes for the e-mail to be captured and indexed in thedatabase 110. The user can also configured the extended memory system106 to index emails as well as other sources of data described herein.The graphical user interface 900 includes a “Send” button 906, whereinupon the user selection of the “Send” button 906, the e-mail istransmitted to the intended recipients.

With reference now to FIG. 10, another exemplary graphical userinterface 1000 that facilitates capture of computer-readable data andindexing of the computer-readable data by corresponding contextual datais illustrated. The graphical user interface 1000 includes text, animage 1002, and a video 1004. In the example shown in FIG. 10, a usercan select a portion 1006 of the text and can set forth some commandthat causes a window 1008 to be presented. The command may be a voicecommand, a right-click on a mouse, or the like. The window 1008 includesa plurality of selectable options that are representative of operationsthat can be performed with respect to the selected text. Theseoperations can include, for example, copy, paste, send, format, etc. Theoperations can also include a “remember” option, which when selected,causes the highlighted text (or other content selected by the user) tobe provided to the extended memory system 106. Selection of the“remember” option can also cause contextual data corresponding to theselected text to be captured and transmitted to the extended memorysystem 106, where it can be used to index the highlighted portion 1006.

Turning now to FIG. 11, an exemplary graphical user interface 1100 thatmay be included in a text editor is illustrated. The text editor may bein a word processing application, in a browser, or the like. In theexample shown in FIG. 11, the user sets forth the text “do you want towatch the bears?” It can be ascertained, as noted above, that the term“bears” as set forth in the text is somewhat ambiguous. To facilitatedisambiguating the term “bears”, a window 1102 can be presented thatdepicts metadata that can be assigned to the text shown in the graphicaluser interface to disambiguate the term “bears”. For example, if theuser intended to refer to the Chicago Bears, the user can select theChicago Bears option in the window 1102, which causes the metadata“Chicago Bears” to be assigned to the text. This metadata may then beprovided to the extended memory system 106, which can index the text bythe entity “Chicago Bears”. Therefore, subsequently, if the user wishesto recollect something about the Chicago Bears and sets forth a queryabout the Chicago Bears, the extended memory system 106 can providerelevant data back to the user (e.g., the message shown in the graphicaluser interface 1100).

FIGS. 12-17 illustrate exemplary methodologies relating to assisting auser with recollecting information about people, places, or things.While the methodologies are shown and described as being a series ofacts that are performed in a sequence, it is to be understood andappreciated that the methodologies are not limited by the order of thesequence. For example, some acts can occur in a different order thanwhat is described herein. In addition, an act can occur concurrentlywith another act. Further, in some instances, not all acts may berequired to implement a methodology described herein.

Moreover, the acts described herein may be computer-executableinstructions that can be implemented by one or more processors and/orstored on a computer-readable medium or media. The computer-executableinstructions can include a routine, a sub-routine, programs, a thread ofexecution, and/or the like. Still further, results of acts of themethodologies can be stored in a computer-readable medium, displayed ona display device, and/or the like.

With reference now to FIG. 12, an exemplary methodology 1200 thatfacilitates indexing computer-readable data by temporally correspondingcontextual data is illustrated. The methodology 1200 starts at 1202, andat 1204, computer-readable data is received from a firstcomputer-executable application. For instance, the firstcomputer-executable application may be a web browser, a socialnetworking application, a word processing application, an image viewingapplication, a video playing application, or other suitable application.At 1206, contextual data about computer-readable data is received from asecond computer-executable application. In an example, the secondcomputer-executable application may be a voice recognition application,and the contextual data may be an identity of the speaker. In anotherexample, the second computer-executable application may be an operatingsystem, and the contextual data can be data acquired from a sensor. At1208, the computer-readable data received at 1204 is indexed by thecontextual data received at 1206. Accordingly, a user can subsequentlyquery a database based upon the contextual data, resulting in retrievalof the computer-readable data. The methodology 1200 completes at 1210.

Now referring to FIG. 13, an exemplary methodology 1300 for constructinga searchable database is illustrated. The methodology 1300 starts at1302, and at 1304, computer-readable data is received from acomputer-executable application. At 1306, data from a sensor on acomputing device is received. The sensor may be a position sensor, acamera, a barometer, a temperature sensor, or the like. At 1308,contextual data about the computer-readable data is identified basedupon the data from the sensor. For instance, the data from the sensormay be an image of a person, and the contextual data about thecomputer-readable data may be an identity of the person. In anotherexample, the contextual data may be the data output by the sensor itselfthus, the contextual data may be the temperature output by thethermometer. At 1310, the computer-readable data received at 1304 isindexed by the contextual data identified at 1308 in a searchabledatabase. The methodology 1300 completes at 1312.

Now referring to FIG. 14, an exemplary methodology 1400 that facilitatesindexing computer-readable data by a disambiguated entity in asearchable database is illustrated. The methodology 1400 starts at 1402,and at 1404 computer-readable data is received from a firstcomputer-executable application. At 1406, an object referenced in thecomputer-readable data is disambiguated. For example, the entitydisambiguator component 206 may disambiguate the object. At 1408,contextual data about the computer-readable data is received from asecond computer-executable application. At 1410, the computer-readabledata received at 1404 is indexed by the disambiguated object referencedin the computer-readable data, and also by the contextual data receivedat 1408. The methodology 1400 completes at 1412.

Now turning to FIG. 15, an exemplary methodology 1500 that facilitatesquerying a database generated by the extended memory system 106 isillustrated. The methodology 1500 starts at 1502, and at 1504, a queryis received that references a context surrounding a person, place, orthing. The context may be a person, a time, a location, etc. At 1506, auser-specific index is searched based upon the contents referenced inthe query. The term “user specific” indicates that the index ispersonalized for the user and is configured to assist the user inrecollecting information people, places, and things that are of interestto the user. At 1508, data is output that identifies the person, place,or thing that had the context corresponding thereto. The methodology1500 completes at 1510.

Now turning to FIG. 16, an exemplary methodology 1600 that facilitatesoutputting a recommendation is illustrated. The methodology 1600 startsat 1602, and at 1604 context of a user is received. This context may belocation of the user, something being reviewed by the user, a meeting tobe attended by the user, or the like. At 1606, a query is constructedbased upon the received context. As described previously, the query maybe constructed automatically and may include a reference to the contextreceived at 1604. At 1608, a search over a user-specific index isperformed based upon the query constructed at 1606. The search mayresult in identification of computer-readable data that may be relevantto the current context of the user. At 1610, content is output as arecommendation based upon the search over the user specific index. Themethodology 1600 completes the 1612.

Now referring to FIG. 17, an exemplary methodology 1700 that facilitatestransmittal of data to the extended memory system 106 is illustrated.The methodology 1700 starts at 1702, and at 1704, a request for memoryassistance is received from the user relative to computer-readablecontent. As noted previously, the request may be spoken command setforth by the user, a gesture performed by the user, a menu selectionperformed by the user, and so on. At 1706, content and associatedcontext is captured responsive to receipt of the request. At 1708, datais transmitted that is indicative of the content and the associatedcontext to a computing system for indexing. Specifically, the data thatis indicative the content and the associated context is transmitted tothe computing system 100, where the extended memory system 106 can indexthe content by the associated context. The methodology 1700 completes at1710.

Various examples are now set forth.

Example 1: A method comprising: receiving computer-readable data from afirst computer-executable application; receiving contextual data aboutthe computer-readable data from a second computer-executableapplication, the contextual data generated independently from thecomputer-readable data; and indexing, in a database, thecomputer-readable data from the first computer-executable application bythe contextual data from the second computer-executable application.

Example 2: The method according to example 1, further comprising:receiving a query, the query references the contextual data; searchingthe database based upon the query; and outputting data that identifiesthe computer-readable data responsive to searching the database.

Example 3: The method according to any of examples 1-2, the databasecustomized for a particular user.

Example 4: The method according to any of examples 1-3, thecomputer-readable data comprises a reference to an entity, the methodfurther comprises: adding metadata to the computer-readable data, themetadata disambiguates the entity; and indexing the computer-readabledata by the metadata.

Example 5: The method according to example 4, further comprisingresponsive to receipt of the computer-readable data, identifying themetadata based upon the contextual data.

Example 6: The method according to any of examples 1-5, furthercomprising: receiving second contextual data from a client computingdevice; formulating a query based upon the second contextual data;identifying the computer-readable data in the database based upon thequery; and transmitting a recommendation to the client computing deviceresponsive to identifying the computer-readable data, the recommendationbased upon the computer-readable data.

Example 7: The method according to any of examples 1-6, wherein thecomputer-readable data is data presented to a person operating acomputing device, and the contextual data is an identity of a secondperson with whom the person is with when the computer-readable data isreceived.

Example 8: The method according to any of examples 1-5, furthercomprising: subsequent to indexing the computer-readable data, receivingsecond contextual data from a client computing device, the secondcontextual data matches the contextual data; formulating a query basedupon the second contextual data; identifying the computer-readable databased upon the query; and transmitting a reminder to the clientcomputing device or another client computing device responsive toidentifying the computer-readable data, the reminder is based upon thecomputer-readable data.

Example 9: The method according to any of examples 1-8, thecomputer-readable data received from a first computing device, thecontextual data received from a second computing device.

Example 10: The method according to any of examples 1-9, furthercomprising: subsequent to indexing the computer-readable data, receivinga query and an identity of a person, the query references the contextualdata; performing a search based upon the query; identifying thecomputer-readable data based upon the search; and responsive toidentifying the computer-readable data, causing the computer-readabledata to be accessible to the person.

Example 11: The method according to example 10, further comprising:subsequent to indexing the computer-readable data, receiving an identityof a third computer-executable application, the person having an accountwith the third computer-executable application; and causing thecomputer-readable data to be accessible to the person by way of thethird computer-executable application.

Example 12: A computing system comprising: a processor; and a memorythat comprises an extended memory system that is executed by theprocessor, the extended memory system, when executed by the processor,configured to: receive computer-readable data from a firstcomputer-executable application executing on a computing device; receivecontextual data that temporally corresponds to the computer-readabledata from a second computer-executable application executing on thecomputing device; and expose the computer-readable data to a searchcomponent, the computer-readable data indexed by the contextual data.

Example 13: The computing system according to example 12 being a datacenter.

Example 14: The computing system according to any of examples 12-13, thecomputer-readable data comprises a reference to an entity, the extendedmemory system comprises an entity disambiguator component that isconfigured to generate metadata to disambiguate the entity, the extendedmemory system configured to index the computer-readable data by themetadata.

Example 15: The computing system according to example 14, the extendedmemory system further comprises a context acquirer component that isconfigured to transmit a signal to a client computing device in responseto receiving the computer-readable data, the client computing deviceconfigured to generate the contextual data and transmit the contextualdata to the computing system responsive to receiving the signal.

Example 16: The computing system according to any of examples 12-15, theextended memory system comprises the search component that is configuredto return the computer-readable data based upon a query that referencesthe contextual data.

Example 17: The computing system according to example 16, the searchcomponent comprises a query disambiguator component that, when the queryincludes a reference to an entity, is configured to disambiguate theentity prior to the search component returning the computer-readabledata.

Example 18: The computing system according to example 16, the searchcomponent is configured to generate the query responsive to receipt ofcontextual data from a client computing device.

Example 19: The computing system according to example 16, the searchcomponent comprises a dialog engine, the dialog engine is configured tooutput suggestions that facilitate navigation through indexed data.

Example 20: A computer-readable storage medium comprising instructionsthat, when executed by a processor, cause the processor to perform actscomprising: indexing computer-readable data by temporally correspondingcontextual data in a database, the computer-readable data output by afirst computer-executable application, the contextual data output by asecond computer-executable application; executing a search over thedatabase based upon a query, the query comprises a reference to thecontextual data; and outputting a search result based upon the search,the search result comprises the computer-readable data.

Referring now to FIG. 18, a high-level illustration of an exemplarycomputing device 1800 that can be used in accordance with the systemsand methodologies disclosed herein is illustrated. For instance, thecomputing device 1800 may be used in a system that supports indexingcomputer-readable data by corresponding contextual data. By way ofanother example, the computing device 1800 can be used in a system thatsupports capturing computer-readable data and corresponding contextualdata. The computing device 1800 includes at least one processor 1802that executes instructions that are stored in a memory 1804. Theinstructions may be, for instance, instructions for implementingfunctionality described as being carried out by one or more componentsdiscussed above or instructions for implementing one or more of themethods described above. The processor 1802 may access the memory 1804by way of a system bus 1806. In addition to storing executableinstructions, the memory 1804 may also store computer-readable data,contextual data, etc.

The computing device 1800 additionally includes a data store 1808 thatis accessible by the processor 1802 by way of the system bus 1806. Thedata store 1808 may include executable instructions, computer-readabledata, contextual data, etc. The computing device 1800 also includes aninput interface 1810 that allows external devices to communicate withthe computing device 1800. For instance, the input interface 1810 may beused to receive instructions from an external computer device, from auser, etc. The computing device 1800 also includes an output interface1812 that interfaces the computing device 1800 with one or more externaldevices. For example, the computing device 1800 may display text,images, etc. by way of the output interface 1812.

It is contemplated that the external devices that communicate with thecomputing device 1800 via the input interface 1810 and the outputinterface 1812 can be included in an environment that providessubstantially any type of user interface with which a user can interact.Examples of user interface types include graphical user interfaces,natural user interfaces, and so forth. For instance, a graphical userinterface may accept input from a user employing input device(s) such asa keyboard, mouse, remote control, or the like and provide output on anoutput device such as a display. Further, a natural user interface mayenable a user to interact with the computing device 1800 in a mannerfree from constraints imposed by input device such as keyboards, mice,remote controls, and the like. Rather, a natural user interface can relyon speech recognition, touch and stylus recognition, gesture recognitionboth on screen and adjacent to the screen, air gestures, head and eyetracking, voice and speech, vision, touch, gestures, machineintelligence, and so forth.

Additionally, while illustrated as a single system, it is to beunderstood that the computing device 1800 may be a distributed system.Thus, for instance, several devices may be in communication by way of anetwork connection and may collectively perform tasks described as beingperformed by the computing device 1800.

Various functions described herein can be implemented in hardware,software, or any combination thereof. If implemented in software, thefunctions can be stored on or transmitted over as one or moreinstructions or code on a computer-readable medium. Computer-readablemedia includes computer-readable storage media. A computer-readablestorage media can be any available storage media that can be accessed bya computer. By way of example, and not limitation, suchcomputer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium that can be used to carry or storedesired program code in the form of instructions or data structures andthat can be accessed by a computer. Disk and disc, as used herein,include compact disc (CD), laser disc, optical disc, digital versatiledisc (DVD), floppy disk, and Blu-ray disc (BD), where disks usuallyreproduce data magnetically and discs usually reproduce data opticallywith lasers. Further, a propagated signal is not included within thescope of computer-readable storage media. Computer-readable media alsoincludes communication media including any medium that facilitatestransfer of a computer program from one place to another. A connection,for instance, can be a communication medium. For example, if thesoftware is transmitted from a website, server, or other remote sourceusing a coaxial cable, fiber optic cable, twisted pair, digitalsubscriber line (DSL), or wireless technologies such as infrared, radio,and microwave, then the coaxial cable, fiber optic cable, twisted pair,DSL, or wireless technologies such as infrared, radio and microwave areincluded in the definition of communication medium. Combinations of theabove should also be included within the scope of computer-readablemedia.

Alternatively, or in addition, the functionally described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-programmable Gate Arrays(FPGAs), Program-specific Integrated Circuits (ASICs), Program-specificStandard Products (ASSPs), System-on-a-chip systems (SOCs), ComplexProgrammable Logic Devices (CPLDs), etc.

What has been described above includes examples of one or moreembodiments. It is, of course, not possible to describe everyconceivable modification and alteration of the above devices ormethodologies for purposes of describing the aforementioned aspects, butone of ordinary skill in the art can recognize that many furthermodifications and permutations of various aspects are possible.Accordingly, the described aspects are intended to embrace all suchalterations, modifications, and variations that fall within the spiritand scope of the appended claims. Furthermore, to the extent that theterm “includes” is used in either the details description or the claims,such term is intended to be inclusive in a manner similar to the term“comprising” as “comprising” is interpreted when employed as atransitional word in a claim.

What is claimed is:
 1. A method executed by at least one computingdevice that is in network communication with a client computing device,wherein the client computing device is configured to execute a firstcomputer-executable application, wherein the first computer-executableapplication outputs first data when the first computer-executableapplication is executed by the client computing device, the methodcomprising: receiving second data output by a second computer-executableapplication, wherein the second computer-executable application outputsthe second data independently of the first computer-executableapplication outputting the first data, and further wherein the firstdata and the second data temporally correspond to one another; indexing,in a database that is accessible to a user of the client computingdevice, the first data by the second data; subsequent to indexing thefirst data by the second data, receiving a query from the clientcomputing device, wherein the query comprises the second data; and inresponse to receiving the query, causing the client computing device topresent the first data due to the first data being indexed by the seconddata in the database.
 2. The method of claim 1, wherein the first datais indexed by the second data in the database in response to the user ofthe client computing device indicating that the first data is to bestored.
 3. The method of claim 1, wherein the client computing deviceexecutes the second computer-executable application.
 4. The method ofclaim 1, further comprising: receiving third data output by a thirdcomputer-executable application, wherein the third computer-executableapplication outputs the third data independently of the firstcomputer-executable application outputting the first data andindependently of the second computer-executable application outputtingthe second data, wherein the third data temporally corresponds to thefirst data; and indexing, in the database, the first data by the thirddata.
 5. The method of claim 4, further comprising: subsequent toindexing the first data by the third data, receiving a second query fromthe client computing device, wherein the second query comprises thethird data; and in response to receiving the second query, causing theclient computing device to present the first data due to the first databeing indexed by the third data in the database.
 6. The method of claim1, further comprising: subsequent to indexing the first data by thesecond data, receiving a second query from the client computing device,wherein the second query comprises the first data; and in response toreceiving the second query, causing the client computing device topresent the second data due to the first data being indexed by thesecond data in the database.
 7. The method of claim 1, wherein the firstdata comprises a reference to an entity, the method further comprising:adding metadata to the first data, the metadata configured todisambiguate the entity; and indexing the first data by the metadatasuch that the first data is indexed by both the second data and themetadata.
 8. The method of claim 1, further comprising: transmitting asignal to the client computing device, wherein the signal is configuredto cause the second computer-executable application to output the seconddata.
 9. The method of claim 1, further comprising: receiving, from theclient computing device, third data; formulating a second query basedupon the third data; identifying the first data based upon the secondquery; and transmitting a recommendation to the client computing devicein response to identifying the first data, wherein the recommendation isbased upon the first data.
 10. A computing system that has a processorand memory operably coupled to the processor, wherein the computingsystem is in network communication with a client computing device thathas a first computer-executable application installed thereon, whereinthe first computer-executable application, when executed by the clientcomputing device, outputs first data, and further wherein the memory hasinstructions stored therein that, when executed by the processor, causethe processor to perform acts comprising: indexing, in a database thatis accessible to the client computing device when operated by a user,the first data by second data, wherein the second data temporallycorresponds to the first data, and further wherein the second data isgenerated by a second computer-executable application independently ofthe first computer-executable application outputting the first data;subsequent to indexing the first data by the second data in thedatabase, receiving a query from the client computing device, whereinthe query comprises the second data; identifying the first data basedupon: the query comprising the second data; and the first data beingindexed by the second data in the database; and in response toidentifying the first data, causing the client computing device topresent the first data to the user.
 11. The computing system of claim10, wherein the client computing device is a mobile telephone or awearable computing device.
 12. The computing system of claim 10, theacts further comprising: subsequent to indexing the first data by thesecond data in the database, receiving a second query from a secondclient computing device, wherein the second query comprises the seconddata, and further wherein the user is operating the second clientcomputing device when the second query is received; identifying thefirst data based upon: the second query comprising the second data; andthe first data being indexed by the second data in the database; and inresponse to identifying the first data, causing the second clientcomputing device to present the first data to the user.
 13. Thecomputing system 10, wherein one of the first computer-executableapplication or the second computer-executable application is a webbrowser.
 14. The computing system of claim 10, wherein one of the firstcomputer-executable application or the second computer-executableapplication is an email application.
 15. The computing system of claim10, wherein one of the first computer-executable application or thesecond computer-executable application is a gaming application.
 16. Thecomputing system of claim 10, wherein the client computing deviceexecutes the second computer-executable application, and further whereinthe second computer-executable application generates the second databased upon sensor data output by a sensor of the client computingdevice.
 17. The computing system of claim 10, wherein the first data isindexed by the second data in response to the client computing devicecapturing a voice signal from the user, wherein the voice signalindicates that the first data is to be indexed in the index.
 18. Acomputer-readable storage medium comprising instructions that, whenexecuted by a processor, cause the processor to perform acts comprising:indexing, in a computer-readable index, first data by second data,wherein the first data is output by a first computer-executableapplication when the first computer-executable application executes on aclient computing device operated by a user, and further wherein thesecond data is output by a second computer-executable applicationindependent of the first computer-executable application outputting thefirst data, wherein the first data and the second data temporallycorrespond to one another; subsequent to indexing the first data by thesecond data in the computer-readable index, receiving a query from theclient computing device, wherein the query comprises the second data;identifying the first data in the computer-readable index based upon:the query including the second data; and the first data being indexed bythe second data in the computer-readable index; and responsive toidentifying the first data, causing the first data to be presented tothe user at the client computing device.
 19. The computer-readablestorage medium of claim 18, wherein the first data is indexed by thesecond data in response to the user indicating that the first data is tobe stored.
 20. The computer-readable storage medium of claim 18, whereinthe first data references an entity, the acts further comprising:assigning metadata to the first data, wherein the metadata is configuredto disambiguate the entity referenced in the first data; and indexing,in the computer-readable index, the first data by the metadata, suchthat the first data is indexed by both the second data and the metadatain the computer-readable index.